🕷️️ Job Radar • SCRAPING

freelancer.com 🟡 2026-05-13

🔹 [Target] Website Data Extraction
👤 Client: 🇦🇪 Sharjah, United Arab Emirates Member since 2026-05-13
💰 Price: $130 Average bid
🚩 Problem: Extracting comprehensive text and images from niche websites requires an efficient, structured approach.
📦 Existing: Not specified

Specifications:

[Method] Python with Scrapy for static content scraping; Selenium for dynamic pages
[UI/UX] Not applicable
[Stack] Scrapy, BeautifulSoup, Selenium, Pandas, Flask (for API if needed)
[Security] Respect [login to view URL]; implement rate limiting and user-agent rotation
[Format] CSV or JSON for text data; image files stored in named folders

Workflow:

1. Analyze website structure using tools like Lighthouse or browser developer console.
2. Develop Scrapy spiders for static content, with fallback to Selenium scripts for dynamic pages.
3. Implement pagination handling and login mechanisms if required.
4. Use BeautifulSoup for parsing HTML and extracting text data (headings, paragraphs, metadata).
5. Download images using Python's requests library or similar tools, storing them in named folders corresponding to their records.
6. Map image filenames/pathnames to text records for easy reference.
7. Implement deduplication strategy based on URL hashes or other unique identifiers.
8. Export data into CSV/JSON format and organize images accordingly.